The present invention is based upon my earlier invention shown in U.S. Pat. No. 5,687,291 with the further improvements of: (1) a principal components analysis of event-averaged potentials to better model the human-cerebral sources, and (2) the inclusion of the attention state in the reliability estimate as determined from eye-movements. In addition, the present invention includes elements from my invention shown U.S. Pat. No. 5,649,061 relating eye fixations and brain wave analysis to decision making, and elements from my invention shown in U. S. Pat. No. 5,689,619 on the use of a fuzzy-logic processor to classify the attention state for display control from eye movement data.
The present invention estimates the decision made by a human as he performs cognitive functions in response to a displayed stimulus. The estimation follows from the corresponding single-event evoked, transient, cerebral potential, where the stimulus is typed by a definite task context. The evoked potential is generated naturally within the brain in response to the occurrence of the external stimulus. The amplitudes and latencies of the waveform components of the evoked potential are influenced by the properties of the stimulus and the mental processing that follows.
The present invention uses this decision estimate to provide real-time, automatic aiding for human interaction with an electronic information processing system when the operator is under a demanding cognitive work-load. The present invention may be used for the unintrusive measurement of cognitive functions as part of a training or testing regimen. Similarly, the present invention may be used to unintrusively monitor the state of a trained human operator for fatigue as part of an electronic safety net to detect degradation in operator performance on computerized tasks. In a more elaborate application, the present invention may be used as a component of an electronic "intelligent" human-computer interface for adaptive automated aiding of cognitive functions.
The present invention applies to computer controlled panel or head mounted video and aural displays used in manned crewstations such as helicopters, aircraft and other manned vehicles; display overlays on video returns in a control station for teleoperations of unmanned remote vehicles and platforms; and displays in communication, command and control stations such as modern air-traffic control or military tactical operations centers. Similarly, the present invention applies to head mounted displays used by an individual who, while netted electronically into a data distribution system, is performing stand-alone tasks such as assembly work, maintenance, or soldiering on the modern battlefield. These include computer controlled visual or aural overlays in head mounted video displays used for virtual reality, stereographics, monocular or binocular vision, and image enhancements for night vision.
There is little prior art for accurately estimating a cognitive decision from the corresponding single-event evoked, transient, cerebral potential. There has been prior work done in the laboratory on the mental control of machines by event averaged evoked cerebral potentials. For example, the amplitude of the P300 component of the event averaged transient potential has been used to select video display cues from a set of randomly repeated cue markers. The transient potential duration is on the order of several 100 milliseconds; however, and event averaging by using repetitive signals demands an unnaturally long attention on the order of 10 seconds by the human operator to the cue markers. The result is a procedure that would interfere with the real-time performance of most tasks.
In another example, the power spectrum of a visually evoked steady state potential has been used to select a visual display cue from a field of display cues with different flash rates. The power spectrum will have a peak at the flash rate for the gazed display cue. However, the power spectrum of the steady state potential is computed from the Fourier transform of a windowed signal of several seconds duration. This process is a form of short term averaging which requires forced visual fixation by the operator on the cue marker and for this reason tends to interfere with task performance.
In still another example reported by G. McMillan in a 1995 publication entitled "Brain-Actuated Control: Thinking ahead to Firefox" (Cseriac Gateway), the changes induced by a subject in the power spectrum of a visually evoked steady state potential, generated while looking toward a 13.25 Hertz flashing light, have been used to control the turn direction of a flight simulator, either left or right. Again, the power spectrum of the steady state potential is computed from the Fourier transform of a windowed signal of several seconds duration, and this short term averaging requires forced concentration by the operator which interferes with task performance. Furthermore, the light source must be intense, flashing (near critical fusion frequency) and within the vision-field of the human subject.
One method for estimating a cognitive decision from the corresponding single-event evoked, transient, cerebral potential follows from the work of S. Cerutti, G. Chiarenza, D. Liberati, P. Mascellani, and G. Pavesi in a 1988 publication entitled "A parametric method of identification of single-trial even-related potentials in the brain" (IEEE Transactions of Biomedical Engineering). These researchers used a parametric method for identifying single-trial, transient, event-related potentials in the brain. Their method assumes a moving average, autoregressive (ARMA) filtering model of the cerebral potential with the stimulating event as an exogenous input. The solution for the cerebral potential is recursively computed from the filtering model where an event average response potential is used as the initial estimate.
U.S. Pat. No. 5,649,061 was awarded to me on an apparatus and method for estimating from the cerebral potential, a decision made for selecting a display icon. The concept of a cerebral source is used with the moving average, autoregressive filter model of Cerutti et al. to parameterize the cerebral potentials. The technique uses an artificial neural network as a decision classifier with inputs consisting of the ARMA coefficients from the Cerutti et al. filter model applied to the transient cerebral potential collected during the visual fixation on the icon, as well as parameters from the visual response. These parameters include the duration of the fixation, the pupil dilation, and the number of eye blinks following the fixation. Essential to the success of the technique is the alignment of the recorded cerebral potential with the start of the fixation and windowing of the data to match the duration of the visual fixation. Lower order decisions of visual recognition and selection are processed during eye fixations.
In a supporting development, my invention shown in U.S. Pat. No. 5,726,916 (TEC JA251) shows an eyetracker that uses electrooculograms to provide the millisecond time resolution of eye movements that is needed to accurately align the fixation with the cerebral potential. This apparatus integrated with the apparatus of U.S. Pat. No. 5,583,795 for an optical eyetracker with head mounted displays, will maintain spatial alignment independent of any shifts of the facemask that is used to hold the electrodes and corrects for long term drifts in skin-surface potential.
The reasoning of Cerutti et al. for the recursive solution process of the moving average, autoregressive model of the cerebral potential is intuitive in nature. They do not provide a stated mathematical basis for their reasoning and computational stability is not necessarily assured by their approach. My invention shown in U.S. Pat. No. 5,687,291 replaces the moving average filter component with a parallel attenuator circuit design to better model the internally located cerebral source. The attenuator values are used as inputs to an artificial neural network decision classifier. The method described in the application has proven to be more effective then that developed by Cerutti et al., at least in computer simulation studies.
In another related development, my U.S. Pat. No. 5,689,619 shows an automated adaptive aider for the control of heads-up displays by eyetracker from display menus. This invention employs a fuzzy logic processor to classify the visual attention state from eye movement data and select a cuing format for the estimated state. This invention automatically provides time and spatial locating cues to aid the human operator as the conflict for attention among multiple tasks becomes more intense.
In the present patent application, the techniques of the above referenced patents are further embodied in a novel design for an automated aider of cognitive functions. The parallel attenuator circuit model of my invention shown in U.S. Pat. No. 5,687,291 is further embodied with the novel use of a set of independent basic waveforms to better represent the internally located cerebral sources generated in response to external stimuli. In this development, the basic waveforms are derived from a principal components analysis of the differences among event-averaged potentials. Also, the decision classifier of my invention shown in U.S. Pat. No. 5,649,061 is further embodied with inputs for the cognitive and visual attention states of the human. In support of this application, the method of U.S. Pat. No. 5,689,619 in which a fuzzy logic processor is used to classify visual attention for aiding display control is expanded to the processing of visual information in general. Finally, the alignment of the cerebral potential record with the visual fixation used in U.S. Pat. No. 5,649,061 is relaxed to enable the processing of higher order decisions beyond those needed for display control.